Method for Determining a State of a Pavement from Surroundings Sensor Data

ABSTRACT

The invention relates to a method/a device for determining a state of a pavement from surroundings sensor data. 
     For determining a state of a pavement from surroundings sensor data, a merger of data received from at least one device that measures a local pavement state or coefficient of friction with data received from a camera ( 4 ) for covering a pavement ( 1 ) extending in front of the vehicle is provided. To this end, when analyzing the camera image data, the locally measured pavement state or coefficient of friction may be assigned to individual image sectors of a camera image whilst taking odometric and time information into account and taken into account for the support and/or plausibilization of an anticipatory and locally resolved coefficient-of-friction estimation or state-of-pavement determination on the basis of camera data.

The invention relates to a method and a device for determining a stateof a pavement from surroundings sensor data, particularly from cameradata.

The detection or determination of the coefficient of friction beingeffective between the tire and the pavement or the detection of thestate of the pavement (e.g., dry, wet, snow-covered and icy), from whichstate of the pavement the coefficient-of-friction group can be derived,is an important prerequisite for assisting the driver with his or herdriving task and thus for avoiding serious accidents or lessening theconsequences thereof. In general, the assessment of road conditionsresides with the driver, who adapts his or her driving behavior to saidroad conditions. Vehicle control systems, such as ESC (ElectronicStability Control)/TCS (Traction Control System) or ABS (Anti-lockBraking System), assist the driver in stabilizing the vehicle in thelimit range so that the driver can perform his or her driving task inextreme situations more easily.

With an increasing degree of driver assistance automation through tohighly automated or autonomous driving, the importance of information onthe state of the pavement or on the coefficient of friction isincreasing continuously. The information on the state of the pavement oron the coefficient of friction is typically used to

-   -   inform the driver    -   warn the driver    -   fix the instants of intervening in the braking system and the        steering gear with driver assistance systems and    -   adjust vehicle control functions (e.g., brake, steering gear).

In driver assistance systems, the avoidance of accidents is getting moreand more important. Emergency braking systems (and the recentlyintroduced emergency steering systems) make an important contributionthereto. However, the effect of such systems decisively depends on thecoefficient of friction of the ground. Moisture, snow and iceconsiderably reduce the coefficient of friction available between thetire and the pavement as against the coefficient of friction availableon a dry pavement.

EP 792 228 B1 shows a system for driving stability control for ESP(Electronic Stability Program)/ESC control systems, by means of whichsystem a coefficient of friction can be determined in specialsituations. When at least one wheel utilizes the coefficient of friction(e.g., when driving on slippery ground), the vehicle brake controlsystem can determine the coefficient of friction from the rotationbehavior of the wheels and from the ESP/ESC acceleration sensors.

DE 102 56 726 A1 shows a method for generating a signal depending on thecondition of the pavement using a reflection signal sensor, e.g., aradar sensor or an optical sensor, thereby making an anticipatorydetection of the state of the pavement in a motor vehicle possible.

DE 10 2004 018 088 A1 shows a pavement detection system with atemperature sensor, an ultrasonic sensor and a camera. The pavement datareceived from the sensors are filtered and compared with reference datain order to determine the practicability of the pavement, wherein thepavement surface (e.g., concrete, asphalt, dirt, grass, sand, or gravel)and the state thereof (e.g., dry, icy, snow-covered, wet) can beclassified.

DE 10 2004 047 914 A1 shows a method for assessing the state of thepavement, in which data received from several different sensors (e.g.,camera, infrared sensor, rain sensor, or microphone) are merged in orderto reach a state-of-pavement classification which a coefficient offriction can be assigned to.

DE 10 2008 047 750 A1 shows a determination of an adhesion coefficientwith few sensors, in which torsional oscillations of a wheel of avehicle are analyzed and a coefficient of friction is estimated on thebasis of this analysis.

DE 10 2009 041 566 A1 shows a method for determining a pavementcoefficient of friction μ, in which a first coefficient-of-frictionparameter, which is constantly updated, and a secondcoefficient-of-friction parameter, which is updated situationally only,are combined with each other in order to obtain a common estimatedfriction value.

WO 2011/007015 A1 shows a laser-based method for coefficient-of-frictionclassification in motor vehicles. To this end, signals of a lidarsensor/CV sensor directed toward the pavement surface are analyzed.After that, a coefficient of friction is assigned, particularly on thebasis of the amplitude of the measured pavement surface. For example,one can estimate whether snow, asphalt or ice form the pavement surface.

WO 2012/110030 A2 shows a method and a device forcoefficient-of-friction estimation by means of a 3D camera, e.g., astereo camera. At least one image of the surroundings of the vehicle isacquired by means of the 3D camera. From the image data of the 3Dcamera, a height profile of the road surface is created in the entirearea extending in front of the vehicle. The expectable local coefficientof friction of the road surface in the area extending in front of thevehicle is estimated from the height profile.

The automatic acquisition of the state-of-pavement information is a keyelement on the way to the realization of autonomous driving in future.

However, the known methods are disadvantageous. On the one hand, theavailability of information is highly limited (ESC). On the other hand,the sensors and algorithms are not sufficiently precise yet (camera, IRsensors, radar) or the robustness of the system is highly insufficientfor safety systems (analysis of torsional oscillations of wheel, stereocamera).

The approach of the inventive solution consists in the followingconsiderations: The coefficient-of-friction information determinedaccording to the state of the art is usually not valid for everypavement segment.

Directly measuring systems are capable of measuring very precisely, butthey are not capable of operating in an anticipatory manner. Typicalexamples of such systems are vehicle control systems, such as ESC, ABSor TCS, which virtually determine the coefficient of friction directlyin the footprint of the tire on the basis of the slipping and running-inbehavior on the tire. Technology-specifically, optical sensors (e.g.,near infrared) also have a very limited capability to deliverinformation in a sufficiently anticipatory manner since the anglerelative to the pavement must not become too acute. Both systems andalso wheel speed analysis are only capable of determining the state ofthe pavement locally.

Other systems, particularly camera/video systems, are only capable ofdetermining the state of the pavement indirectly (e.g., by means ofclassification) and are therefore less precise than directly measuringsystems for process-related reasons. However, systems having a coveragedepth of some/several meters (e.g., 1 m-20 m, 2 m-100 m, 5 m-200 mdepending on the design of the camera) and having a width that issufficient for pavement surface detection are particularly well suitedfor an extensive detection of the pavement extending in front of thevehicle due to their actual application as surroundings sensors or frontcameras.

An inventive method for determining a state of a pavement fromsurroundings sensor data provides a merger of data received from atleast one device (sensor) that measures a local coefficient of frictionor determines state-of-pavement information on the vehicle and/orparticularly on or directly in front of at least one vehicle wheel/tirewith data received from a camera or from a larger-range vehiclesurroundings sensor that covers the pavement extending in front of thevehicle. To this end, when analyzing the camera image data, the locallymeasured pavement state or coefficient of friction or the localstate-of-pavement information may be assigned to individual imagesectors of a camera image whilst taking odometric and time informationinto account and taken into account for the support and/orplausibilization of an anticipatory and locally resolvedcoefficient-of-friction estimation or state-of-pavement determination onthe basis of camera data.

In this connection, odometric information is information thatcharacterizes a motion of the vehicle and particularly comprises vehiclesensor system data, such as measured quantities of a chassis, of a powertrain, of a steering gear, as well as measured quantities of anavigation device of the vehicle. Thus, a performed motion or trajectoryof the vehicle can be determined whilst taking the time information intoaccount, or a future trajectory can be predicted in combination with thesurroundings sensor system.

In particular, while the vehicle is in motion, a limited pavementsegment that at first is only covered by the camera and whose pavementstate or coefficient of friction is estimated from camera images can bemeasured afterwards when the locally measuring sensor is moving over it.On the basis of this measured coefficient of friction or this determinedstate-of-pavement information, pavement segments extending in front ofthe vehicle can now be assessed, the image analysis for said pavementsegments extending in front of the vehicle leading to similar oridentical results as the image analysis for the original limitedpavement segment (in previous camera images).

The coefficient of friction, also called adhesion coefficient orcoefficient of static friction, indicates what force can be maximallytransmitted between a pavement surface and a vehicle tire (e.g., in thetangential direction). Thus, said coefficient is an essential measure ofthe state of the pavement. Aside from the state of the pavement,properties of the tire must be taken into account in order to be able todetermine the coefficient of friction completely. Typically, onlystate-of-pavement information is taken into account for an estimation ofthe coefficient of friction (e.g., from camera image data) since it isgenerally impossible to determine any tire properties from camera imagedata.

In other words, systems locally determining the coefficient of frictionor the state of the pavement (such as ESC including ABS/TCS) or ananalysis of the torsional oscillations of the wheel (both on the basisof the wheel speed signal) and/or optical sensors (e.g., infrared/lasersensors) for determining the pavement surface or measuring the localcoefficient of friction are merged with the camera/video sensor systemfor the extensive detection of the pavement extending in front of thevehicle such that the discrete measuring points of the locally measuringdevice can be tracked on the basis of odometric and time information(e.g., on the basis of the vehicle motion in the camera image) and canthus be easily assigned to the individual image sectors (pavementsegments) for the purpose of the support and plausibilization of thecamera algorithms.

The inventive method for determining the state of the pavement ensures avery precise, high-resolution and, above all, anticipatory determinationof the locally resolved pavement state or coefficient of friction. Asagainst predetermined classification methods of astate-of-pavement/coefficient-of-friction estimation from camera imagedata alone, the inventive method has proven to be particularly adaptablesince the actually measured local coefficients of friction or determinedlocal pavement information make—due to the assignment to the currentcamera image data—the method largely resistant to disturbances thatmight occur when the camera covers the pavement, whereby the safetysystems of the vehicle can be prepared for pavement states predictivelyand situationally or the driver can be informed/warned.

In an advantageous embodiment, image analysis includes an assignment ofa locally measured coefficient of friction to a pavement segment in atleast one camera image if the consideration of odometric and timeinformation reveals that the pavement state/coefficient of friction ofthis pavement segment has been locally measured afterwards. Inparticular, a pavement segment can be determined from the camera imageby segmentation, wherein segmentation preferably delivers segmentshaving comparable pavement states. One can determine from the odometricand time information which pavement segment from a camera image wasdriven over afterwards and what local coefficient of friction wasmeasured in doing so or what local pavement state was determined indoing so.

Preferably, image analysis provides a classification of individualpavement segments in camera images on the basis of particular features.In particular, said particular features may be assigned to predeterminedstates of the pavement. The determined state of the pavement (e.g., dry,wet, snow-covered, icy) is an indicator of the coefficient of frictionavailable between the tire and the pavement. A class of pavementsegments (in which the same state of the pavement was determined fromthe camera image) can now be assigned to a coefficient of frictionlocally measured afterwards or a pavement state locally determinedafterwards, whereby an anticipatory coefficient-of-friction estimationfor all pavement segments assigned to this class can be performedsuccessfully.

According to an advantageous embodiment, the camera image is subdividedinto a two-dimensional grid in the plane of the pavement and the atleast one measured local coefficient of friction or pavement state isassigned to at least one cell of the grid.

To this end, a representation of the pavement surface imaged by thecamera may be created, said representation showing the distances on thepavement surface true to scale (e.g., a bird's eye view), wherein thedistances between all grid lines of the grid would be fixed in ahorizontal direction or a vertical direction.

Alternatively, a grid could be superimposed on the camera image, saidgrid reflecting the perspective distortion of the surroundings of thevehicle (and of the pavement) by the camera, whereby the contents ofeach grid cell could correspond to an equally sized pavement segmentwith real distances.

Preferably, the number of the cells of the grid is determined by thehomogeneity of the pavement or pavement surface, particularly in thecamera image. If the camera image shows a largely homogeneous pavementsurface, one can use a smaller number of grid cells than with aninhomogeneous pavement surface. Different pavement surface materials,puddles, snow-covered surfaces and leaves may cause inhomogeneouspavement surfaces, on which the state of the pavement, and thus thecoefficient of friction, may change very quickly.

Advantageously, the number of the cells of the grid is determined by thecurrent driving situation and/or the criticality thereof. In criticaldriving situations, a larger number of cells may be used to make thelocally resolved state-of-pavement/coefficient-of-friction estimationfrom the camera image even more precise, whereby, for example, thecontrol of the brakes for an emergency braking maneuver can be optimizedwhilst taking local state-of-pavement/coefficient-of-friction changesinto account.

Furthermore, the number of the cells of the grid may be determined bythe computing power available for image analysis.

Possibly, the number of cells may be reduced to 1. As a rule, however, aplurality of cells is to be provided for the grid in order to make localresolution possible.

According to a preferred embodiment, the result of the analysis of thecamera data is predictively applied afterwards, whilst taking themeasured state-of-pavement/coefficient-of-friction data assigned to thecamera image into account, to a subsequently acquired camera image. Saidassignment is preferably performed on the basis of cells having the sameor similar features with respect to the state of the pavement, wherein,in particular, a pavement state/coefficient of friction confirmed ormade plausible on the basis of a locally measured coefficient offriction or a locally determined pavement state may be assigned toindividual cells belonging to a common class.

Advantageously, a vehicle corridor is calculated from a predictedmovement trajectory of the vehicle, by means of which vehicle corridorthe positions of the individual locally measuring sensors and of thevehicle wheels can be predictively assigned to pavement segmentsextending in front of the vehicle in the camera image (i.e., inparticular, to individual cells of a grid). The movement trajectories ofthe vehicle can be predicted from vehicle sensor data and/or fromsurroundings sensor data (camera, radar, lidar, etc.) in a manner knownper se.

Advantageously, a class probability is assigned to individual pavementsegments or grid cells. For example, one may indicate that a particularcell is to be assigned to class 1 at 80% and to another class at 20%,whereby the fact that there may actually be sectors representing varyingconditions within one cell can be taken into account. For example, 60%of a contents of a cell may represent a wet pavement and 40% mayrepresent a dry pavement.

Preferably, a monocular camera is used as a camera sensor. Mono camerasare well-established driver assistance cameras and cheaper than stereocameras.

According to an advantageous embodiment, a stereo camera is used as acamera sensor. As against a mono camera, a stereo camera resolves theimage data spatially. Depending on the respective requirements, bothimages or only one of the two images may be analyzed forstate-of-pavement/coefficient-of-friction estimation.

In a preferred realization, an optical sensor is used, exclusively or inaddition to other sensors, as a locally measuring device/sensor. Theoptical sensor is preferably directed toward the pavement surface andcan locally determine the three-dimensional shape of the pavementsurface, wherefrom the state of the pavement can be derived or acoefficient of friction can be estimated.

Alternatively, ultrasonic or radar sensors may be used as such localmeasuring devices as long as they are capable of locally determining thethree-dimensional shape of the pavement surface.

In a particularly advantageous embodiment, at least one measuring devicemeasuring and/or deriving coefficients of friction from the speedsignals of a vehicle wheel (R1-R4) is used, exclusively or in additionto other devices, as a locally measuring device. The slip of the tireand the oscillations of the tire can be analyzed from the wheel speedsignal and the coefficient of friction can be classified on the basisthereof. For example, DE 10 2008 047 750 A1 shows such an analysis ofthe oscillation behavior of the tire, from which behavior a spectrum ofstimulation by the pavement is determined, said spectrum correlatingwith the coefficient of friction.

ABS/ESC/TCS systems being capable of measuring or deriving maximumcoefficients of friction from the speed signals of individual vehiclewheels by an analysis of periods of increasing slip may also be used assuch measuring devices.

The invention also relates to a device for determining a state of apavement. The device comprises a camera, at least one device designed tomeasure a local coefficient of friction or to determine a local pavementstate, and a camera data analysis device. The latter is designed to takethe locally measured coefficient of friction/state of the pavement intoaccount during camera data analysis. The camera data analysis device isdesigned to assign the locally measured coefficient of friction/state ofthe pavement to individual image sectors of a camera image whilst takingodometric and time information into account and to be capable of takingthe locally measured coefficient of friction/state of the pavement intoaccount for the support and/or plausibilization of an anticipatory andlocally resolved coefficient-of-friction estimation or state-of-pavementdetermination on the basis of camera data.

In the following, the invention will be explained in greater detail onthe basis of figures and exemplary embodiments, in which

FIG. 1 shows a camera image of a region of the surroundings of thevehicle extending in front of the vehicle;

FIG. 2 shows a bird's eye view representation of the scene shown in thecamera image;

FIG. 3 shows a subdivision of a part of the representation into cells bymeans of a grid, in which individual cells are classified; and

FIG. 4 shows a vehicle with locally measuring sensors in a grid, inwhich individual cells are classified.

FIG. 1 shows, by way of example, a camera image of a region of thesurroundings of the vehicle extending in front of the vehicle, whichimage was acquired by a front camera of a moving vehicle. Camera-baseddriver assistance functions can be realized on the basis of the sameimage, e.g., Lane Departure Warning (LDW), Lane KeepingAssistance/System (LKA/LKS), Traffic Sign Recognition (TSR), IntelligentHeadlamp Control (THC), Forward Collision Warning (FCW), PrecipitationDetection, Adaptive Cruise Control (ACC), Park Assist, Emergency BrakeAssist (EBA), or Emergency Steering Assist (ESA).

The camera image shows a pavement (1) with a largely homogeneoussurface. One can see lane markings on the surface: A continuous sideline marking the left end of the pavement and a continuous side linemarking the right end of the pavement as well as center line segments(3) of the broken/dashed central pavement marking. The pavement (1)could be an asphalt or concrete pavement. One can see a puddle (2) onthe pavement (1).

FIG. 2 shows a bird's eye view representation of the scene shown in thecamera image in FIG. 1. This representation can be determined from thecamera image, wherein, if a mono camera is used, imaging properties ofthe camera (4), the installation geometry of the camera in the vehicle(5), the actual height of the vehicle (due to tire positioncontrol/chassis control), the pitch angle, the yaw angle and/or the rollangle are preferably taken into account. One can assume that thepavement surface is even.

If a stereo camera is used, the representation can be directlydetermined due to the acquired 3D image data, wherein further aspectsmay be taken into account in this case as well.

The representation is essentially characterized by the fact thatdistances shown in the representation correspond to real distances,i.e., the median strip segments shown are arranged equidistantly on thereal pavement as well.

The representation in FIG. 2 shows the pavement (1), the puddle (2) andthe center line segments (3) of the pavement marking, which are alreadyincluded in the camera image (FIG. 1). The representation additionallyincludes a vehicle (5) with a camera (4), wherein the image in FIG. 1was acquired by means of the camera (4). The dashed arrow indicates thepredicted trajectory (T) of the vehicle (5). For this straight-aheadmotion, the distance traveled s along the trajectory (T) in the case ofa uniform motion with a velocity v can be determined from s=vt whilsttaking the information about time t into account. In this manner, onecan determine, whilst taking the odometric and time information intoaccount, when, e.g., the left front wheel of the vehicle (5) will reachthe puddle (2).

This representation does not show the local coefficient-of-frictionmeasuring devices of the vehicle (5) that are capable of measuring thepavement state and/or the coefficient of friction between the tire andthe pavement surface (1) locally (i.e., essentially below the vehicle).Such devices may be optical sensors (6) directed toward the pavementsurface (1) (infrared or laser sensors in particular) or devices such asESC that determine the available coefficient of friction locally on thewheels or that derive a coefficient of friction from an analysis of awheel speed signal.

FIG. 3 shows how a grid (G) is superimposed on a part of therepresentation in FIG. 2, whereby the representation is subdivided intoindividual cells. By means of classification, individual classescontaining a piece of information about the pavement state or thecoefficient of friction may be assigned to individual grid cells. In thepresent example, those cells in which the pavement is a dry asphaltpavement are assigned to class K1. Those cells in which wet asphalt ispresent are assigned to class K2. Cells in which a different type ofground is present may be assigned to class Kn.

For example, a simple classification consists in a subdivision of thepavement segments into four classes: dry asphalt (K1), wet asphalt (K2),snow, and ice. More generally, n classes K1 to Kn may be provided thatmay take, e.g., various pavement materials (asphalt, concrete, sand,gravel, etc.) and various condensate states (e.g., dry, wet, snow, ice)into account. Among the n classes one may also provide a remaining classfor pavement segments that cannot be assigned to any of the other(predetermined) classes.

The number of the grid cells or the size of an individual cell of thegrid (G) may be varied. If the camera image shows a largely homogeneouspavement surface (such as the pavement (1) except for the puddle (2) inthe present case), one can use a smaller number of grid cells than witha pavement surface that is inhomogeneous on the whole or than in theregion of the puddle (2). Different pavement surface materials, puddles(2), snow-covered surfaces and leaves may cause inhomogeneous pavementsurfaces, on which the coefficient of friction may change very quickly.In critical driving situations, a larger number of cells may also beused to make the locally resolvedstate-of-pavement/coefficient-of-friction estimation from the cameraimage even more precise, whereby, for example, the control of the brakesfor an emergency braking maneuver can be optimized whilst taking localcoefficient-of-friction changes into account. Finally, the number of thecells of the grid (G) may be determined by the computing power availablefor image analysis.

FIG. 4 shows a grid-based assignment of local measuredcoefficient-of-friction data to classified pavement segments or gridcells.

As explained on the basis of FIG. 3, the individual cells are firstclassified on the basis of particular features by image analysis bymeans of image-processing algorithms, wherefrom class values (K1 to Kn)for the cells located in front of the vehicle result.

For example, FIG. 4 shows a pavement situation comparable to thepavement situation shown in FIG. 2 and FIG. 3. Most cells are assignedto class K1, whereas a 2×2 block of cells is assigned to class K2. Theinformation that there is a puddle or a coherent wet area (2) (on anotherwise dry pavement (1)) in front of the vehicle can be inferred fromcamera analysis and classification, but it is impossible to assign anactual pavement state/coefficient of friction to these classes (K1, K2,. . . ) if no further knowledge (average values, empirical values, etc.)is available. The material of the dry pavement (1) could be asphalt withgood grip or smooth asphalt. The puddle (2) could be shallow or deep,could be a water or oil puddle, etc.

This uncertainty can be eliminated by measuring the pavementstate/coefficient of friction by means of a local sensor of the vehicle(5). The vehicle (5) shown has an optical sensor (6) and four measuringdevices that derive/measure, as local sensors, coefficients of frictionfrom the speed signals of one of the four vehicle wheels (R1-R4) at atime. These local sensors measure a current pavement state/coefficientof friction that in each case can be assigned to that cell of the grid(G) within which the local sensor measures the pavementstate/coefficient of friction. In FIG. 4, said cells are those cells inwhich the optical sensor (6) measures and in which one wheel (R1-R4) ata time is in contact with the pavement (1).

Classes K1 to Kn and the local measured values are combined with eachother on the basis of odometric and time data.

The subdivision into classes on the basis of the camera data can now bemerged, on the basis of odometry when the vehicle is driving over, withthe results of the local sensors with the results of cameraclassification in the respective cells. In FIG. 4, for example, thatcell in which the optical sensor (6) measures the pavementstate/coefficient of friction was assigned to class K1 (dry asphalt). Onthe basis of the measured value, a coefficient of friction or a state ofthe pavement can be assigned to this class for the first time or apre-estimated pavement state/coefficient of friction for this class (K1)can be made plausible, corrected or validated as a merged pavementstate/coefficient of friction (K1). In the cells on the left (left rearwheel (R3) and right rear wheel (R4)), a corresponding mergedcoefficient of friction or state of the pavement (K1) was alreadydetermined when the vehicle was driving over.

Similarly, a measured value for class K2 (wet asphalt) can be obtainedin that cell in which the right front wheel (R2) of the vehicle islocated. To this end, the measuring device derives/measures the pavementstate/coefficient of friction from the speed signal of the right frontwheel (R2) of the vehicle and a merged coefficient of friction/state ofthe pavement (K2) for class K2 is determined therefrom. This cell wasalready assigned to class K2 in a previously acquired camera image, andit is determined, on the basis of the odometric and time data, if andwhen the local sensors measure in the region of these classified cells.

Afterwards, the merger results obtained in this manner are predictivelyapplied to the currently anticipatory camera image and assigned to thecorresponding classes of the individual cells in said camera image.

Alternatively, the future vehicle corridor can be calculated from thepredicted movement trajectory (T) of the vehicle (5). This is also shownin FIG. 4. For example, one can estimate on the basis of the currentcamera image that the optical sensor (6) will be able to measure thepavement state/coefficient of friction in those cells through which thedashed trajectory (T) extends, and the point in time thereof can beestimated from the odometric and time data. In the case of a uniformmotion with a constant angular velocity u along a circuit having aradius r, the distance s traveled during a period of time t isdetermined from s=rωt. Thus, the positions of the optical sensor (6) andof the individual wheels (R1-R4) of the vehicle (5) can be predictivelyand precisely assigned to the cells in front of the vehicle with theirfeatures. One can see, for example, that the optical sensor (6) willsoon be able to measure the pavement state/coefficient of friction intwo cells assigned to class K2 and that a merged coefficient of friction(K2) for class 2 will thus be available in the third cell and in thefourth cell to the right of the cell with the optical sensor (6).

LIST OF REFERENCE NUMERALS

-   1 pavement/pavement surface-   2 puddle-   3 center line segment-   4 camera-   5 vehicle-   6 optical sensor for local pavement state determination-   T (movement) trajectory-   G grid-   K1 class 1-   K2 class 2-   Kn class n-   K1 merged coefficient-of-friction estimation for class 1-   K2 merged coefficient-of-friction estimation for class 2-   R1 left front wheel-   R2 right front wheel-   R3 left rear wheel-   R4 right rear wheel

1. A method of determining a state of a pavement on which a vehicle isdriving, comprising merging locally measured data received from at leastone locally measuring device that measures a local pavement state orcoefficient of friction of the pavement, with camera data received froma camera (4), and performing an image analysis comprising analyzing thecamera data, wherein the locally measured data represents the localpavement state or coefficient of friction which is respectively assignedto individual image sectors of a camera image in the camera data whilsttaking odometric information of the vehicle and time information intoaccount, and is taken into account for support and/or plausibilizationof an anticipatory and locally resolved coefficient-of-frictionestimation or state-of-pavement determination based on the camera data.2. The method according to claim 1, wherein the image analysis includesassigning the local pavement state or coefficient of friction to atleast one pavement segment in the camera image if consideration of theodometric information and the time information reveals that a predictedpavement state or coefficient of friction of this pavement segment hasbeen locally measured in the locally measured data thereafter.
 3. Themethod according to claim 1, wherein the image analysis provides aclassification (K1, K2, . . . , Kn) of individual image sectors orpavement segments in the camera image based on particular features. 4.The method according to claim 1, wherein the camera image is subdividedinto a two-dimensional grid (G) in a plane of the pavement and the localpavement state or coefficient of friction is assigned to a cell of thegrid (G).
 5. The method according to claim 4, wherein a total number ofthe cells of the grid (G) is determined by a homogeneity of the pavement(1).
 6. The method according to claim 4, wherein a total number of thecells of the grid (G) is determined by a current driving situationand/or a criticality thereof.
 7. The method according to claim 4,wherein a total number of the cells of the grid (G) is determined by anavailable computing power for performing the method.
 8. The methodaccording to claim 1, wherein a result of the image analysis of thecamera data is predictively applied afterwards, whilst taking the localpavement state or coefficient of friction assigned to the camera imageinto account, to a subsequently acquired camera image.
 9. The methodaccording to claim 1, further comprising calculating a vehicle corridorfrom a predicted movement trajectory (T) of the vehicle (5), by means ofwhich vehicle corridor positions of individual wheels (R1-R4) of thevehicle (5) and/or of a locally measuring sensor of the at least onelocally measuring device are predictively assigned to pavement segmentsin the camera image, said pavement segments extending in front of thevehicle.
 10. The method according to claim 1, further comprisingassigning probability values to an image sector or a pavement segment,said probability values indicating with what probability the imagesector or the pavement segment is to be assigned to a first class and toat least a second class.
 11. The method according to claim 1, wherein amono camera is used as the camera (4).
 12. The method according to claim1, wherein a stereo camera is used as the camera (4).
 13. The methodaccording to claim 1, wherein a sensor or an optical sensor (6) is usedas the locally measuring device, said sensor or said optical sensor (6)locally determining a three-dimensional shape of a pavement surface ofthe pavement.
 14. The method according to claim 1, wherein at least onemeasuring device that measures and/or derives a local coefficient offriction from speed signals of a vehicle wheel (R1-R4) is used as thelocally measuring device.
 15. A device for determining a state of apavement on which a vehicle is driving, comprising a camera (4) thatprovides camera data, at least one locally measuring device configuredto measure a local pavement state or coefficient of friction of thepavement, and a camera data analysis device configured to take the localpavement state or coefficient of friction into account during cameradata analysis of the camera data, wherein, the camera data analysisdevice is further configured to assign the local pavement state orcoefficient of friction to individual image sectors of a camera image ofthe camera data whilst taking odometric information of the vehicle andtime information into account and to take the local pavement state orcoefficient of friction into account for support and/or plausibilizationof an anticipatory and locally resolved coefficient-of-frictionestimation or state-of-pavement determination based on the camera data.16. A method of determining a surface condition of a driving surface onwhich a vehicle is driving, comprising steps: a) with a camera on saidvehicle, producing camera data including a camera image of a selectedsurface area of said driving surface ahead in front of said vehicle; b)performing an image analysis of said camera image using an imageanalysis algorithm to determine an estimated surface conditioncomprising an estimated pavement state or an estimated coefficient offriction of said selected surface area; c) driving said vehicle forwardwhereby said selected surface area comes into a sensing range of alocally measuring sensor on said vehicle, and using time information andodometric information of said vehicle to achieve sensing registration ofsaid locally measuring sensor with said selected surface area; d) withsaid locally measuring sensor, sensing locally measured datarepresenting an actual surface condition comprising an actual pavementstate or an actual coefficient of friction of said selected surfacearea; e) comparing said actual surface condition with said estimatedsurface condition, and dependent on any discrepancy therebetweenupdating said image analysis algorithm in a manner that would reducesaid discrepancy; and f) repeating said steps a) to e), using saidupdated image analysis algorithm in said step b), with regard to asubsequent selected surface area of said driving surface ahead in frontof said vehicle.